Papers with execution-based scoring
SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. |
| Approach: | They propose a syntactic probing framework for detecting and quantifying such contamination in large language models. |
| Outcome: | The proposed framework generates syntactic variants of test queries for four widely used NL2SQL datasets. |